Tests for Assessment of Agreement Using Probability Criteria

For the assessment of agreement using probability criteria, we obtain an exact test, and for sample sizes exceeding 30, we give a bootstrap-t test that is remarkably accurate. We show that for assessing agreement, the total deviation index approach of Lin (2000, Statist. Med., 19, 255–270) is not consistent and may not preserve its asymptotic nominal level, and that the coverage probability approach of Lin et al. (2002, J. Am. Stat. Assoc., 97, 257–270) is overly conservative for moderate sample sizes. We also show that the nearly unbiased test of Wang and Hwang (2001, J. Stat. Plan. Inf., 99, 41–58) may be liberal for large sample sizes, and suggest a minor modification that gives numerically equivalent approximation to the exact test for sample sizes 30 or less. We present a simple and accurate sample size formula for planning studies on assessing agreement, and illustrate our methodology with a real data set from the literature.